Method and system for control of a digital camera system

ABSTRACT

Embodiments of the invention include methods, systems, and software for control of digital camera systems. In one embodiment, a digital camera system includes a lens and an image sensor. The digital camera system is rotatable about a single axis, and the image sensor has more pixel sensors in a first orientation substantially parallel to the single axis than in a second orientation perpendicular to the first orientation. Additionally, an application, coupled to the digital camera system via a wireless network, is to control rotation of the digital camera system.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of application Ser. No. 16/557,683,filed Aug. 30, 2019, which is hereby incorporated by reference.

TECHNICAL FIELD

Embodiments of the invention relate to the field of digital cameras and,more specifically, to control of a digital camera system.

BACKGROUND ART

Digital camera systems have gained wide usage in various industries. Forexample, they are used in applications such as surveillance, videoconferencing, live production, lecture capture, and distance learning.

A pan-tilt-zoom (PTZ) camera is a popular option for digital camerasystems. A PTZ camera may perform panning, i.e., swiveling the camerahorizontally from a fixed position; it may perform tilting, i.e.,rotating the camera up/down in a vertical plane from the fixed position;and it may perform zooming, i.e., adjusting the focal length of the lensof the camera. A PTZ camera may be controlled remotely, allowing anoperator to monitor a large area and to pinpoint an area of interestwithin by panning, tilting, and zooming the camera to the area. Theflexibility of the PTZ camera, however, comes with a high manufacturingcost.

SUMMARY

Embodiments of the disclosed invention include camera systems. In oneembodiment, a camera system includes a lens and an image sensor. In oneembodiment, the camera system is rotatable about a single axis and theimage sensor has more pixel sensors in a first orientation substantiallyparallel to the single axis than in a second orientation perpendicularto the first orientation. Additionally, an application, coupled to thecamera system via a wireless network, controls rotation of the camerasystem. In one embodiment, a single assembly of a camera system includesboth lens and image sensor, and the single assembly is switchablebetween a first orientation and a second orientation perpendicular tothe first orientation.

Embodiments of the disclosed invention include methods to controldigital camera systems. Images are obtained using a lens and an imagesensor of a camera system. In one embodiment, the camera system isrotated about a single axis, where an application, coupled to the camerasystem via a wireless network, is to control the rotation of the camerasystem. Additionally, the image sensor is oriented to have more pixelsensors in a first orientation substantially parallel to the single axisthan in a second orientation perpendicular to the first orientation.

Embodiments of the disclosed invention include non-transitorymachine-readable storage media for control of a digital camera system.In one embodiment, a non-transitory machine-readable storage mediumprovides instructions that, when executed, cause a mobile device toobtain images using a lens and an image sensor of a camera system;rotate the camera system about a single axis, where an applicationwithin the mobile device is to control the rotation of the camerasystem; and orient the image sensor to have more pixel sensors in afirst orientation substantially parallel to the single axis than in asecond orientation perpendicular to the first orientation.

Embodiments of the disclosed invention provide ways for control of adigital camera system. Because the digital camera system has a singleaxis about which the camera system can rotate, and the image sensor hasmore pixel sensors in the orientation substantially parallel to thesingle axis, the digital camera system has a greater coverage in theorientation without relying on fine-grained rotations in multiple axes.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention may best be understood by referring to the followingdescription and accompanying drawings that are used to illustrateembodiments of the invention. In the drawings:

FIG. 1 illustrates a multi-axis camera.

FIG. 2 illustrates a camera system according to one embodiment of theinvention.

FIG. 3A illustrates the overlay of an image circle of a lens and animage sensor placed horizontally.

FIG. 3B illustrates the overlay of an image circle of a lens and animage sensor placed vertically.

FIG. 4A illustrates a first vertical field of view when an image sensoris placed horizontally.

FIG. 4B illustrates a second vertical field of view when an image sensoris placed vertically.

FIG. 5A illustrates a physical layout of a camera system according toone embodiment of the invention.

FIG. 5B illustrates a partial cross-section view of a camera systemaccording to one embodiment of the invention.

FIG. 5C illustrates a set of exemplary image sensor formats.

FIG. 6 illustrates the operations of a camera system according to oneembodiment of the invention.

FIG. 7 is a flow diagram illustrating the operations at an electronicdevice for control of a camera system according to some embodiments ofthe invention.

FIG. 8 is a block diagram illustrating an electronic device that mayimplement a camera application according to one embodiment of theinvention.

DETAILED DESCRIPTION

Aspects of the present disclosure are directed to methods andapparatuses for control of a digital camera system. In the followingdescription, numerous specific details such as logic implementations,resource partitioning/sharing/duplication implementations, types andinterrelationships of system components, and logicpartitioning/integration choices are set forth in order to provide amore thorough understanding of the present invention. It will beappreciated, however, by one skilled in the art that the invention maybe practiced without such specific details. In other instances, controlstructures, gate level circuits, and full software instruction sequenceshave not been shown in detail in order not to obscure the invention.

Bracketed text and blocks with dashed borders (e.g., large dashes, smalldashes, dot-dash, and dots) may be used herein to illustrate optionaloperations or components that add additional features to embodiments ofthe invention. However, such notation should not be taken to mean thatthese are the only options or optional operations, and/or that blockswith solid borders are not optional in certain embodiments of theinvention.

Terms

References in the specification to “one embodiment,” “an embodiment,”“an example embodiment,” etc., indicate that the embodiment describedmay include a particular feature, structure, or characteristic, butevery embodiment may not necessarily include the particular feature,structure, or characteristic. Moreover, such phrases are not necessarilyreferring to the same embodiment. Further, when a particular feature,structure, or characteristic is described in connection with anembodiment, it is submitted that it is within the knowledge of oneskilled in the art to affect such feature, structure, or characteristicin connection with other embodiments whether or not explicitlydescribed. A “set,” as used herein, refers to any positive whole numberof items including one item.

In the following description and claims, the terms “coupled” and“connected,” along with their derivatives, may be used. It should beunderstood that these terms are not intended as synonyms for each other.“Coupled” is used to indicate that two or more elements, which may ormay not be in direct physical or electrical contact with each other,co-operate or interact with each other. “Connected” is used to indicatethe establishment of communication between two or more elements that arecoupled with each other.

In the following description and claims, different terms about shapes,orientations, and directions are used. It should be understood thatthese terms cover shapes, orientations, and directions substantiallysimilar/close to the shapes, the orientations, and the directions.Substantially similar/close means the deviation to the used shapes(e.g., rectangle/oblongs, circles, and spheres), orientations (e.g.,portrait and landscape), and directions (e.g., parallel andperpendicular) is within, e.g., 5% of the respective values. Forexample, an orientation parallel to an axis means the orientation iswithin 4.5 degrees of the absolute parallel between the orientation andthe axis. Additionally, a shape such as rectangle as described in thisdescription includes a shape with round or sharp corners.

A digital camera is a camera that captures images and/or videos indigital memory such as the machine-readable storage media discussedherein relating to the definition of an electronic device. While amobile device often incorporates a digital camera, the digital cameradescribed in this Specification is a digital camera that is physicallyseparated from a mobile device unless specified otherwise. A digitalcamera is an electronic device and it may include a network interfacefor communicating with another electronic device through a communicationnetwork. Unless specified otherwise, the digital camera (or simplyreferred to as camera) in this Specification has the network interfaceand is identifiable with one or more IP addresses.

Multi-Axis Digital Camera Systems

A multi-axis camera system is a popular choice for many applicationssuch as surveillance, video conferencing, live production, lecturecapture, and distance learning. FIG. 1 illustrates a multi-axis camera.The multi-axis camera 100 is coupled to a mount 116 that supports and/ormaneuvers the multi-axis camera 100. The mount 116 may be set on astable surface or other hardware (e.g., tripod) to control its movement.It is assumed that the mount 116 and/or the stable surface forms aplacement plane for the multi-axis camera to be placed on. Themulti-axis camera 100 may rotate about multiple axes. As shown, it mayperform panning at reference 102, i.e., swiveling the camerahorizontally from its fixed position about a first axis that isperpendicular to the placement plane. The panning may be about the firstaxis that is located at or near the center of the multi-axis camera 100.The panning may be driven by a motor within the multi-axis camera 100and/or movement of the mount 116.

Additionally, the multi-axis camera 100 may perform tilting at reference104, i.e., rotating the camera up/down in a vertical plane from a fixedposition, and the axis for tilting is parallel to the placement planefor the multi-axis camera. The tilting may also be driven by a motorwithin the multi-axis camera 100 and/or movement of the mount 116.

Some embodiments of the multi-axis camera 100 may also perform zooming,i.e., adjusting the focal length of the lens of the camera. Thesemulti-axis cameras are referred to as pan-tilt-zoom (PTZ) cameras, andthey are commonly used in various applications.

Camera Systems in Embodiments of the Invention

While a PTZ camera system offers great flexibility, the flexibilitycomes with costs and simpler camera systems may be implemented. FIG. 2illustrates a camera system according to one embodiment of theinvention. A camera 202 is communicatively coupled to cloud storage 262and a controlling electronic device 220 via a communications network260.

Each of the camera 202, the cloud storage 262, the controllingelectronic device 220, and the communication network 260 may bemanufactured, owned, and/or operated by a different party. For example,the communication network 260 is typically owned and operated by atelecom service provider, and it includes one or more wireless and/orwireline networks. The cloud storage 262 is typically owned and operatedby a cloud service provider, and it may store and process images/videoscaptured by the camera 202.

The camera 202 is typically manufactured by a camera system vendor,which may also provide the controlling electronic device 220 and thecamera application 222 within. Alternatively, the controlling electronicdevice may be manufactured by another party (e.g., a smartphone vendor),while the camera application 222 is provided by the camera systemvendor. The system 200 may be implemented and maintained by an operator(e.g., a surveillance system operator and/or a video conferencingprovider).

The camera 202 includes camera hardware 250 and camera controller 210.The camera hardware 250 includes lens 252, one or more motors and/orswitch 254, and image sensor 256. The lens 252 may be a fixed focallength lens that has the ability to be focused for different distances.Alternatively, the lens 252 may also be a fixed focus lens that isintended for use at a single, specific working distance. One or more ofthe motors and/or switch 254 control the rotation of the camera 202 as awhole to rotate about a single axis (e.g., panning) Note that unlessspecified otherwise, in this Specification, switching/switchable refersto a rotation between two orientations, i.e., the operation has only twopossible options (e.g., a binary operation between portrait andlandscape orientations); while rotating/rotatable refers to a change oforientations with multiple alternative orientations (e.g., portrait,landscape, 45 degrees or other angles between the two, etc.). A switchmay be a mechanical one (e.g., a knob) or an electronic one (e.g., usinga motor), which causes the rotation between the two orientations.

In some embodiments, one or more additional motors/switch control therotation of the image sensor 256. In some embodiments, a single assemblymodule (or simply assembly) includes both of the lens 252 and the imagesensor 256, and the lens 252 and the image sensor 256 rotate togetherwhen the assembly is rotated/switched. The same assembly including lens252 and image sensor 256 may also include other components of thecamera, but the assembly is one component of the camera (i.e., not thewhole camera), which may include components not rotatable by therotation/switching of the single assembly. Such design is advantageousas it provides consistency between the lens 252 and the image sensor 256in image capturing after rotation. For example, when the image sensorrotates while the lens does not, the image quality may deteriorate sincethe alignment between the image sensor and the lens may be broken afterthe rotation of the image sensor. With the same assembly including bothlens and image sensor, the alignment between the lens and image sensorremains intact regardless the orientation of the image sensor.

The camera controller 210 controls movements of the camera 202 orcomponents of the camera 202. The camera controller 210 includes a lensand/or image sensor controller 201 and a single axis plane motioncontroller 204. The lens and/or image sensor controller 201 causes therotation of the lens and/or image sensor (e.g., the assembly describedabove). The single axis plane motion controller 204 causes the rotationof the camera 402 as a whole about a single axis (e.g., rotating thecamera casing/housing about a mount axis). Additionally, the cameracontroller 210 may also include an image analyzer 206, which determinesthe coverage of the object to be monitored in real-time. The imageanalyzer may be additionally or alternatively implemented in the cameraapplication 222.

The image analyzer 206 may identify features from captured images/videoswithout user input. For example, the image analyzer 206 may identify thelocations of different parts of one or more human bodies to determine ifthe current image sensor orientation is sufficient to capture the humanbodies. The image analyzer 206 may do so by identifying outlines offacial features, points of the facial features (“facial featurepoints”), arms, hands, and/or feet. The identified facial features mayinclude mouth, eyes, eyebrows, nose, irises, pupils, teeth, lips,cheeks, hair, and T-zone (an area including the nose and across theforehead). If the image analyzer 206 determines that the coverage isinsufficient to capture a human body (for example, it may determine thatpart of the face of a human body is not captured in an image), it maynotify the single axis plane motion controller 204 and lens and/or imagesensor controller 201 to cause the rotation of the camera 202 and/or therotation of the image sensor until the human body or part thereof isfully included in a captured image, sequence of images, or video.

The image analyzer 206 may use techniques/tools such as artificialintelligence (AI), machine learning, and/or data visualization todetermine the coverage of the objects to be monitored. In machinelearning, a supervised learning algorithm may be used to identify amodel through empirical learning to transform a set of inputs into a setof outputs. For example, a subset of inputs with corresponding outputsmay be provided to the model to train the model. Once the model isidentified, the model may be used to predict output for subsequentinputs. The model may include a plurality of coefficients, which whenused in conjunction with system inputs, provide an output that isconsistent with observed behaviors or status. The machine learningproblems, such as identifying coverage of the objects to be monitored,may be cast as a convex optimization problem and solved by standardoptimization tools.

The camera controller 210 coordinates with an application within thecontrolling electronic device 220 to control the operations of thecamera hardware 250. The camera application 222 may be an applicationwithin a mobile device, in which case the application is a mobileapplication. A mobile application is also referred to as mobile softwareapplication, mobile app, or simply app and it is a software applicationdesigned to run on a mobile device. A mobile application runs directlyon a mobile device (rather than through a web browser) once downloadedto the mobile device from an app store (also referred to as appmarketplace). Alternatively, the camera application may be a desktop ora web application, through which an operator may control the camera 202,similar to the mobile application.

The camera application 222 includes a camera rotator 223, whichcoordinates with the single axis plane motion controller 204 to rotatethe camera 202 as a whole (e.g., including casing and/or mount). Thecamera application 222 may also include a lens and image sensor rotator224, which coordinates with the lens and/or image sensor controller 401to rotate/switch the lens and/or image sensor.

Additionally, the camera application 222 may include a graphics userinterface (GUI) 225. The GUI 225 may be a visual guide for a user of thecamera application 222 to control the camera 202 (e.g., presented on adisplay device of electronic device 220). For example, the GUI 225 mayinclude an icon for a switch, operating on which causes the image sensorto rotate from one orientation to another. The GUI 225 may include anicon for image sensor rotation/switching, operating on which causes theimage sensor to rotate/switch. The GUI 225 may present images capturedby the camera 202, and the user of the camera may use a (virtual orphysical) control panel or a touch screen to control the rotations ofthe camera, image sensor, and/or the lens. The user may use gestures orvoice commands to control the rotations. For example, the user may usefinger taps or swipes of one or more fingers on the touch screen tocontrol rotations. The user may also issue voice commands that may beinterpreted by camera application 222 to control the rotations. Forexample, the camera application 222 may incorporate features of avirtual assistant (e.g., smart speakers), so that a user may speak tothe controlling electronic device 220 with phrase such as the following:“OK, Camera, please rotate the sensor to a vertical position,” whichcauses the image sensor of the camera 202 to rotate to the verticalorientation.

In one embodiment, the orientation of the image sensor may be controlledby the screen orientation of the mobile device 220 in which the cameraapplication 222 is included. The two screen orientations of the mobiledevice that are the most common for the user to view the capturedimages/videos from the camera 202 include the portrait screenorientation and landscape screen orientation. In the portrait/verticalscreen orientation, the mobile device is placed upright, with the heightof the display screen is greater than the width (i.e., the mobile deviceis placed vertically). In the landscape/horizontal screen orientation,the mobile device is placed horizontally, with the width of the displayscreen is greater than the height. These orientations will be describedfurther with reference to FIG. 6. By changing the orientation of themobile device's screen, the user may cause the image sensor to berotated to the same orientation. Thus, when the mobile device is rotatedto a vertical orientation, the image sensor may follow and rotate to thevertical orientation. When the mobile device is rotated to a horizontalorientation, the image sensor may follow and rotate to the horizontalorientation.

Each of user gestures (fingers and mobile device rotation) and voicecommands may cause the camera application 222 to send a singleinstruction (e.g., a message) to the camera 202, which causes the lensand/or image sensor controller 201 to perform rotation of the imagesensor (and optionally the lens) to rotate from one orientation (e.g.,landscape orientation) to another (e.g., portrait orientation). With thevarious ways that the camera application 222 controls the camera 202,the user may control the rotations of the camera 402 remotely with greatflexibility.

Image Sensor Orientations and Their Impact on Coverage

The embodiments of the invention control rotation of an image sensor ina camera system, and such rotation affects the coverage of the camerasystem significantly. A lens projects an image onto an image sensor, andthe size of the projected image is based on the design of a lens. Forexample, in some implementations, a ⅔-inch lens has an image circle ofaround 11 mm and a ½-inch lens has an image circle of around 8 mm. Alens is said to cover an image sensor when its image circle produces animage over the full sensor size. FIG. 3A illustrates the overlay of animage circle of a lens and an image sensor placed horizontally. Theimage circle of the lens at reference 350 covers the image sensor placedhorizontally at reference 342. An image sensor is placed horizontallywhen its longer sides are parallel to the placement plane of the camerawhile its shorter sides are perpendicular to the placement plane (i.e.,a landscape orientation).

The area under surveillance of a camera may be larger than the imagecircle. As shown in the figure, the surveillance area 370 is larger thanthe size of the image sensor in both horizontal (left-right) andvertical (top-bottom) directions. To monitor the surveillance area 370,a camera may use panning to cover the horizontal direction and tiltingto cover the vertical direction. That is, to monitor the surveillancearea 370 with the image sensor placed horizontally, a multi-axis camerasuch as a PTZ camera can be used.

FIG. 3B illustrates the overlay of an image circle of a lens and animage sensor placed vertically. FIG. 3B is similar to FIG. 3A, but theimage sensor is placed vertically at reference 344, where its shortersides are parallel to the placement plane of the camera while its longersides are perpendicular to the placement plane (i.e., a portraitorientation).

When the image sensor is placed vertically and the camera is used tomonitor the same area, the camera no longer needs to use tilting as theheight of the image sensor is sufficient to cover the height of thesurveillance area 370. The camera, however, may perform additionalpanning (in comparison to the landscape orientation) since now it coversless horizontally when stationary. Yet such a drawback is reasonablebecause the multi-axis camera is no longer needed, and a simpler camerais sufficient to monitor the same surveillance area.

The impact of image sensor orientation on the field of view (FOV) of acamera is significant. The field of view is an observable range anobserver (camera in this case) may see. For example, the angular fieldof view (AFOV) of a lens is related to its focal length (f) and verticaldimension (h) of the image sensor, and it may be calculated using thefollowing formula:

$\begin{matrix}{{AFOV} = {2 \times {\tan^{- 1}\left( \frac{h}{2\; f} \right)}}} & (1)\end{matrix}$

Let's assume the focal length of the lens is 3.6 mm and the image sensorhas the dimensions of 4.8 mm and 3.6 mm (exemplary values for a ⅓-inchimage sensor). For a horizontally placed image sensor, the AFOV isaround 53 degrees, using the formula (1) above and plugging in h=3.6 mmand f=3.6 mm. On the other hand, if the same image sensor is orientedvertically, the AFOV is around 67 degrees. These image sensororientations differ by 14 degrees in the angular field of view, whichsignificantly changes the height of covered area, as shown in FIGS.4A-B.

FIG. 4A illustrates a first vertical field of view when an image sensoris placed horizontally/in a landscape orientation. The image sensor 450is rectangular in shape and it has a width 454 longer than its height452. The image sensor 450 may work with a lens 422 to captureimages/videos of an object 430. When the image sensor 450 is placedhorizontally as shown at reference 402, its height 452 is parallel tothe lens 422 on a plane perpendicular to the placement plane of thecamera containing both image sensor and lens. The camera is placed at aplacement height 412 and it captures images of the object 430 at aworking distance 442. The angular field of view (AFOV) for the settingis AFOV1 at reference 414.

FIG. 4B illustrates a second vertical field of view when an image sensoris placed vertically/in a portrait orientation. FIG. 4B is similar toFIG. 4A but the image sensor 450 is placed vertically instead as shownat reference 404. Thus, the longer sides of the image sensor 450 (thesides along the image sensor width) are parallel to the lens 422 on aplane perpendicular to the placement plane. The AFOV for this setting isAFOV2 at reference 416.

As shown, the difference between AFOV1 and AFOV2 causes the differenceof object coverage by the camera. The difference of object coverage canbe illustrated using the exemplary ⅓-inch image sensor and the 3.6 mmfocal length lens. As calculated above relating to formula (1) above,the AFOV1 is 53° and AFOV2 is 67°. Using a working distance is 1.5meters, a common setting for surveillance or conferencing, the verticalheight of the object 430 covered by the setting in FIG. 3A is around 1.5meters (2×1.5×tan (53/2)≈1.5). With the vertical field of view being 1.5meters (less than five feet), it is insufficient to capture the entireheight of an average adult human For example, when the placement heightis 0.8 meters, the camera will not capture an image of the head and feetof the human body as indicated at reference 432. For applications suchas surveillance, missing the head (showing theemotion/intent/identifiable features of the human) and feet (showing themovement of the human) is a significant shortcoming that makes thecamera unsuitable for these applications, unless tilting is incorporatedto cover the missing portion of the human body. In other words, amulti-axis camera is required to offer the sufficient coverage when theimage sensor is placed horizontally.

In comparison, the vertical height covered by the setting in FIG. 4B(portrait orientation) is around 1.99 meters with the AFOV2 being 67°and the working distance 1.5 meters (2×1.5×tan (67/2)≈1.99). Thevertical field of view of 1.99 meters (around six-foot-five-inch or6′5″) is sufficient to capture the entire height of an average adulthuman as indicated at reference 434. For reference, the average heightfor an adult male in the U.S. is around five-foot-nine-inch (5′9″) andan adult female five-foot-four-inch (5′4″) as shown in data compiled inthe National Health and Nutrition Examination Survey (NHANES) conductedfrom 2007 to 2010.

With the sufficient coverage using the vertically placed image sensor,the camera does not need to perform tilting to capture an image of thefull height of typical human bodies. That is, the multi-axis camera isno longer required to offer the sufficient coverage vertically. Theadvantage of not requiring tilting is significant, as it means that thecamera may rotate about only a single axis (e.g., performing panning)and still offers the coverage required, as shown in FIG. 3B. In otherwords, with the image sensor placed vertically, a simpler camera (whichsupports panning but not tilting) may be sufficient for an applicationsuch as surveillance.

The simpler camera (e.g., camera 202) will cost less to design,manufacture, and maintain since the rotation of the whole camera aboutone axis is removed. In one embodiment, the image sensor is placedstatically in the portrait orientation, in which case the only rotationby the camera is panning. In another embodiment, the image sensororientation can be dynamically rotated into the portrait orientationfrom another orientation (e.g., the landscape orientation).

The rotation of the image sensor is often simpler than tilting. Forexample, the rotation of the image sensor can be implemented throughswitching, where only two orientations are allowed (the portrait andlandscape orientation). The rotation of the image sensor (withouttilting of the whole camera) can also be steadier and/or slower thantilting of the whole camera. A commercial PTZ camera may tilt at thespeed between 0.1° to 60° per second, and the tilting speed depends onvarious factors such as the object's movement and the vertical coveragethe camera is required to obtain. The variable tilting speeds that a PTZcamera supports makes the PTZ camera challenging to design, manufacture,and maintain, since one or more motors with granular motion controlsneed to be included to control the tilting of the PTZ camera. Thus, thePTZ camera can be expensive.

In contrast, the rotation of the image sensor may be done onlyperiodically and once it's in the desired orientation, it may stay inthe orientation for some time (minutes or even hours instead ofseconds). Comparing to tilting, the rotation of the image sensor issemi-static since it may be done only periodically without real-timeorientation changes of orientation in tilting. The simpler rotation ofthe image sensor can be realized using a switch (e.g., switching betweentwo pre-determined orientations) or a coarser-grained motor (coarsercomparing to a motor for tilting). By using such simpler image sensorrotation, embodiments of the invention cost less to manufacture andmaintain thus the resulting camera can be sold cheaper and be deployedin more applications.

FIG. 5A illustrates a physical layout of a camera system according toone embodiment of the invention. Camera 502 has a lens 512, a casing514, and a mount 516. The mount 516 helps the lens 512 to rotate about asingle axis. The image sensor 522 is included in the casing 514, thusnot shown without a cross-section view 515. FIG. 5B illustrates apartial cross-section view 515 shows a single assembly module thatincludes both lens 512 and image sensor 522 according to one embodimentdiscussed herein above.

The orientation reference shows different orientations. A first axis isperpendicular to the placement plane of the camera 502. The axis may belocated at the center of the camera 502, and it may be between thecenter of the top of the casing 514 and the center of the bottom of thecasing 514. Alternatively, it may be another position about which thecamera 502 is to perform panning The first axis may be referred to asthe top-bottom axis, which is parallel to a plane vertical/perpendicularto the placement plane, where the plane may thus be referred to as thevertical plane of the camera 502. A second axis is parallel to both theplacement plane and the vertical plane, and it intersects with the firstaxis. Since it passes through the left and right side of the camera 502,it may be referred to as the left-right axis. A third axis is parallelto the placement plane but perpendicular to the vertical plane. Thethird axis intersects with and is perpendicular to both the first andsecond axis. Since it passes through the front and back of the camera502, it may be referred to as the front-back axis.

The camera 502 may perform panning and rotate about the top-bottom axis.The panning causes the camera casing 514 to rotate. Yet the cameracasing 514 can't perform tilting and rotate about the left-right axis,but instead the image sensor 522 (and lens 512 when both are included ina single assembly) may rotate about the front-back axis so that theimage sensor may be placed vertically and extend its coveragevertically.

In some embodiments, the camera casing 514 may not rotate about thetop-bottom axis (i.e., no panning) Instead, the image sensor rotatesabout the front-back axis to increase the vertical coverage of thecamera 502. In these embodiments, the camera casing 514 does not rotatein any axis, yet the image sensor may be rotated/switched to achieve theoptimal field of view. For example, the image sensor 522 and the lens512 may be on a single assembly 520, and the single assembly 520 isrotatable/switchable between a first orientation and a secondorientation perpendicular to the first orientation. In other words, theimage sensor 522 may be rotatable/switchable between the twoorientations (e.g., landscape and portrait orientations) regardlesswhether the camera casing 514 itself may be rotatable on other axes. Thecamera 502 may include a motor to cause rotation/switch. Alternatively,or additionally, the camera 502 may include a mechanical switch to causethe rotation/switch.

In these embodiments, the switch/rotation of the single assembly 520 mayalso be controlled by the controlling electronic device 220 using acamera application 222 as discussed herein.

FIG. 5C illustrates a set of exemplary image sensor formats. A digitalcamera uses a lens with a diaphragm to focus light onto an image sensor.The lens may be a fixed focal length lens that has the ability to befocused for different distances, and it may also be a fixed focus lensthat is intended for use at a single, specific working distance.

An image sensor (also referred to as imager) is a sensor that detectsand conveys information used to make an image. The image sensor may be asemiconductor charge-coupled devices (CCD) sensor, Quanta image sensor(QIS), or an active pixel sensor in complementarymetal-oxide-semiconductor (CMOS) or N-type metal-oxide-semiconductor(NMOS, Live MOS) technologies. Embodiments of the invention apply toimage sensors implemented in different technologies. Note that an imagesensor may have an active area that is smaller than the size of theimage sensor, and it is the area of the image sensor on which image isformed in a given mode of a camera. For simplicity of discussion, it'sassumed that the active area is substantially equal to the size of theimage sensor; and when they are not equal, embodiments of the inventionconcern the active area of the image sensor and the orientation of theactive area as the active area is where the image is formed.

The image sensor format of a camera is the shape and size of an imagesensor. The image sensor format determines the angle of view of aparticular lens when used with a particular sensor. Sensor size of animage sensor is often expressed as optical format in inches (oftenshorthanded using the double prime as shown in the figure) ormillimeters (mm). The exemplary sensors in the figure include a 1-inchsensor 562, a ⅔-inch sensor 564, a ½-inch sensor 566, and a ⅓-inchsensor 568. Each image sensor includes numerous sensors for pixels (alsoreferred to as picture elements or pels) to capture images/videos, andthese numerous sensors are referred to as pixel sensors. The pixelsensors are typically distributed evenly on the image sensor to arriveat a uniform image resolution, but embodiments of the invention do notrequire the uniformity of the pixel sensor distribution on the imagesensor.

Each sensor has a respective aspect ratio. An aspect ratio of ageometric shape is the ratio of its size in different dimensions. Theimage sensors of a camera are often in the shape of a rectangle (alsoreferred to as an oblong to exclude a square as a type of rectangle).While typical aspect ratios include 4:3, 3:2, and 16:9, as shown atreference 560 in FIG. 5C, embodiments of the invention apply to otheraspect ratios for image sensors. Note that an image sensor may have anactive area that is smaller than the size of the image sensor. Theactive area is the area of the image sensor on which image is formed ina given mode of a camera. For simplicity of discussion, it's assumedthat the active area is substantially equal to the size of the imagesensor.

FIG. 6 illustrates the operations of a camera system according to oneembodiment of the invention. In this embodiment, a camera system 602includes a lens 612, a casing 614, a mount 616, an image sensor 622,which is not visible in the view, but is shown at two differentorientations as the image sensor placed horizontally at reference 626and the image sensor placed vertically at reference 628. The imagesensor's orientation can be switched between the two orientationsthrough a switch or a motor, either of which may be configurable by amobile device as shown at reference 650.

A single axis about which the camera system 602 is rotatable is shown atreference 690, and the rotation of the camera system 602 about thesingle axis, where the rotation may be caused by the mount 616 or amotor within the camera system 602 shown at reference 696. Forreference, the axis directions are shown as the direction parallel tothe single axis at reference 692, and the direction perpendicular to thesingle axis at reference 694.

As shown, the controlling electronic device to control the camera system602 is a mobile device, which includes a mobile application 630. Themobile application 630 is similar to the camera application 222, and itmay include the camera rotator 223, lens and image sensor rotator 224,GUI 225, and optionally the image analyzer 206. The mobile device may beplaced in a portrait screen orientation or a landscape screenorientation, as shown at references 632 and 634, respectively. Themobile device may detect its own orientation, and the application 630obtains the detected orientation, and causes the image sensor torotate/switch to the orientation accordingly. Note as explained atreference 618, an image analyzer may be implemented in either the camerasystem 602 or the mobile application 630.

The mobile device may control the rotation of the camera system 602and/or the rotation/switch of the image sensor 622 (and the lens 612when both are in a single assembly) through (1) screen rotation, and (2)GUI as shown at reference 660. These operations are discussed in moredetails relating to FIG. 2.

Operations per Some Embodiments

The operations in the flow diagrams may be described with reference tothe exemplary embodiments of the other figures. However, it should beunderstood that the operations of the flow diagrams can be performed byembodiments of the invention other than those discussed with referenceto the other figures, and the embodiments of the invention discussedwith reference to these other figures can perform operations differentthan those discussed with reference to the flow diagrams.

FIG. 7 is a flow diagram illustrating the operations at an electronicdevice for control of a camera system according to some embodiments ofthe invention. The electronic device may be a controlling electronicdevice 220 such as a mobile device in an embodiment, and it may remotelycontrol a camera or camera system such as the ones at references 202,502, or 602. The camera system may include a lens, an image sensor, ahousing, and/or a mount. The lens may be a fixed focal length lens thathas the ability to be focused for different distances, and it may alsobe a fixed focus lens that is intended for use at a single, specificworking distance.

At reference 702, images are obtained using the lens and the imagesensor of the camera system. At reference 704, the camera system isrotated about a single axis, wherein an application, coupled to thecamera system via a wireless network, is to control the rotation of thecamera system. At reference 706, the image sensor is oriented to havemore pixel sensors in a first orientation substantially parallel to thesingle axis than in a second orientation perpendicular to the firstorientation.

In one embodiment, the image sensor is to be fixed in an orientation(e.g., vertical orientation) so that the active area of the image sensorhas the longer sides perpendicular to the placement plane of the camerasystem.

In one embodiment, the image sensor having more pixel sensors in thefirst orientation is caused by the electronic device to control a switchwithin the camera system to rotate the image sensor (e.g., to the firstorientation), and the rotation by the switch is independent from therotation of the camera system. In one embodiment, the switch has twopre-defined orientations to choose from, thus the switch operation maybe to switch the image sensor either from a horizontal orientation to avertical orientation, or from the vertical orientation to the horizontalorientation. In one embodiment, the switch may be a mechanical mechanismwithout using an electronic motor. In that case, a user of the camerasystem may operate on the switch directly without using the electronicdevice.

In one embodiment, the image sensor having more pixel sensors in thefirst orientation is caused by the electronic device to control a motorwithin the camera system to rotate the image sensor (e.g., to the firstorientation), and the rotation by the motor is independent from therotation of the camera system. Note that the rotation of the camerasystem as a whole may be also caused by a set of motors, but that set ofmotors are likely to be finer-grained motors than the motor to rotatethe image sensor as the former needs to perform operations such aspanning and they may be operated in a variety of speeds.

In one embodiment, the application is within a mobile device, andwherein the application includes a graphics user interface (GUI) throughwhich an operator causes the image sensor to rotate (e.g., from thefirst orientation to the second orientation and vice versa). In oneembodiment, the mobile device switching to a landscape screenorientation causes the image sensor to be placed horizontally, andwherein the mobile device switching to a portrait screen orientationcauses the image sensor to be placed vertically.

In one embodiment, an image analyzer determines an optimal orientationof the image sensor based on detection of an object under surveillance,and the determination of the optimal orientation causes the image sensorto rotate to the optimal orientation. For example, the image analyzerdetermines where is the face of a human body under surveillance, andwhether the image sensor is in the optimal orientation to cover thewhole human body. If the image analyzer determines that a part of theface is cut off in the current image sensor orientation, it maydetermine that the other image sensor orientation is the optimalorientation and cause the image sensor to rotate to the optimalorientation.

In one embodiment, a single instruction from the application causes theimage sensor to rotate ninety degrees. In one embodiment, theapplication is within a mobile device, and a finger swipe or tap on themobile device causes the camera system to rotate. In one embodiment,another finger swipe to tap on the mobile device causes the image sensorto rotate.

In one embodiment, the camera system comprises a single assemblyincluding both lens and image sensor, and the image sensor having morepixel sensors in the first orientation substantially parallel to thesingle axis than in the second orientation perpendicular to the firstorientation is caused by rotating the single assembly.

The operations and the camera system discussed herein may be widely usedin surveillance, video conferencing, live production, lecture capture,distance learning, and other variety of applications. In someembodiments of the invention, the image sensor is fixed in oneorientation (e.g., the vertical orientation) so that the image sensorhas more pixel sensors in the orientation substantially parallel to asingle axis than in a second orientation perpendicular to the firstorientation, where the camera system is rotatable about the single axis.In some embodiments, the image sensor is rotatable, and the rotation ofthe image sensor is independent from the rotation of the camera system.

As explained herein, the rotation of the camera system about the singleaxis (e.g., panning) may use a set of fine-grained motors, while therotation of the image sensors may use a coarse-grained motor or aswitch. That is, the rotation of the image sensors replaces the rotationof the camera system about another axis (e.g., tilting) using anotherset of fine-grained motors. Since the coarse-grained motor or switch ischeaper than the fine-grained motors, the resulting camera systems costless to manufacture and maintain, thus they can be sold cheaper and bedeployed in more applications that what the PTZ cameras may be deployed.

In some camera systems, the lens and image sensors are included in asingle assembly, and the single assembly is switchable between a firstorientation and a second orientation perpendicular to the firstorientation. When the switching between the two orientations iscontrolled by an application (e.g., the camera application 222), thesecamera systems may achieve a sufficient coverage without resorting tothe set of fine-grained motors (e.g., for panning or tilting). That is,the rotation of the image sensors replaces the rotation of the camerasystem as a whole in any axes. Without any set of fine-grained motors,these camera systems can cost even less.

In other words, the camera systems in embodiments of the inventionreduce hardware requirements to rotation/switch, and thus may be cheaperto manufacture and/or easier to maintain than the traditional PTZcameras.

Operating Environments and Implementations

An electronic device stores and transmits (internally and/or with otherelectronic devices over a network) code (which is composed of softwareinstructions and which is sometimes referred to as computer program codeor a computer program) and/or data using machine-readable media (alsocalled computer-readable media), such as machine-readable storage media(e.g., magnetic disks, optical disks, solid state drives, read onlymemory (ROM), flash memory devices, phase change memory) andmachine-readable transmission media (also called a carrier) (e.g.,electrical, optical, radio, acoustical, or other form of propagatedsignals—such as carrier waves, infrared signals). Thus, an electronicdevice (e.g., a computer) includes hardware and software, such as a setof one or more processors (e.g., of which a processor is amicroprocessor, controller, microcontroller, central processing unit,digital signal processor, application specific integrated circuit(ASIC), field programmable gate array (FPGA), other electroniccircuitry, a combination of one or more of the preceding) coupled to oneor more machine-readable storage media to store code for execution onthe set of processors and/or to store data. For instance, an electronicdevice may include non-volatile memory containing the code since thenon-volatile memory can persist code/data even when the electronicdevice is turned off (when power is removed). When the electronic deviceis turned on, that part of the code that is to be executed by theprocessor(s) of the electronic device is typically copied from theslower non-volatile memory into volatile memory (e.g., dynamicrandom-access memory (DRAM), static random-access memory (SRAM)) of theelectronic device. Typical electronic devices also include a set of oneor more physical network interface(s) (NI(s)) to establish networkconnections (to transmit and/or receive code and/or data usingpropagating signals) with other electronic devices. For example, the setof physical NIs (or the set of physical NI(s) in combination with theset of processors executing code) may perform any formatting, coding, ortranslating to allow the electronic device to send and receive datawhether over a wired and/or a wireless connection. In some embodiments,a physical NI may comprise radio circuitry capable of (1) receiving datafrom other electronic devices over a wireless connection and/or (2)sending data out to other devices through a wireless connection. Thisradio circuitry may include transmitter(s), receiver(s), and/ortransceiver(s) suitable for radiofrequency communication. The radiocircuitry may convert digital data into a radio signal having the properparameters (e.g., frequency, timing, channel, bandwidth, and so forth).The radio signal may then be transmitted through antennas to theappropriate recipient(s). In some embodiments, the set of physical NI(s)may comprise network interface controller(s) (NICs), also known as anetwork interface card, network adapter, or local area network (LAN)adapter. The NIC(s) may facilitate in connecting the electronic deviceto other electronic devices allowing them to communicate with wirethrough plugging in a cable to a physical port connected to a NIC. Oneor more parts of an embodiment of the invention may be implemented usingdifferent combinations of software, firmware, and/or hardware.

A wireless network (also referred to as a cellular network) is a networkof devices communicating using radio waves (electromagnetic waves withinthe frequencies 30 KHz-300 GHz). A wireless communication may followwireless communication standards, such as new radio (NR), LTE (Long-TermEvolution), LTE-Advanced (LTE-A), wideband code division multiple access(WCDMA), High-Speed Packet Access (HSPA), WiFi (wireless fidelity), andBluetooth. Furthermore, the communications between the devices such asnetwork devices and mobile devices in the wireless communication networkmay be performed according to any suitable generation communicationprotocols, including, but not limited to, the first generation (1G), thesecond generation (2G), 2.5G, 2.75G, the third generation (3G), thefourth generation (4G), 4.5G, the fifth generation (5G) communicationprotocols, and/or any other protocols either currently known or to bedeveloped in the future.

A communications network includes one or more wireless and/or wirelinenetworks, and it allows a user of a digital camera system to control oneor more digital cameras remotely across networks such as wide areanetworks (WANs), metropolitan area networks (MAN), local area networks(LANs), internet area networks (IANs), campus area networks (CANs), andvirtual private networks (VPNs). Entities in the communicationsnetworks, such as mobile devices and digital cameras, may be givenInternet Protocol (IP) addresses for identification, and the IPaddresses may be IP Version 4 (IPv4) or IP Version 6 (IPv6).

A mobile device may access a wireless communication network and receiveservices from the wireless communication network. A mobile device may bea user equipment (UE), which may be a subscriber station, a portablesubscriber station, a mobile station (MS), or an access terminal (AT).The terminal device may be one of a mobile phone, a cellular phone, asmart phone, a tablet, a wearable device, a personal digital assistant(PDA), a portable computer, an image capture terminal device such as agaming terminal device, a music storage and playback appliance, avehicle-mounted wireless terminal device, a smart speaker, a set-topbox, and a customer premise equipment (CPE). Note that while smartphones are often used as examples of mobile devices in thisspecification, embodiments of the invention apply to other mobiledevices as well.

FIG. 8 is a block diagram illustrating an electronic device that mayimplement a camera application according to one embodiment of theinvention. The electronic device 800 may be the controlling electronicdevice 220 in some embodiment. The electronic device 800 can includemany different components. These components can be implemented asintegrated circuits (ICs), portions thereof, discrete electronicdevices, or other modules adapted to a circuit board such as amotherboard or add-in card of a computing system, or as componentsotherwise incorporated within a chassis of the computing system. Notealso that electronic device 800 is intended to show a high-level view ofmany components of the computing system. However, it is to be understoodthat additional components may be present in certain implementations andfurthermore, different arrangement of the components shown may occur inother implementations.

In one embodiment, the electronic device 800 includes a processor 801, anon-transitory machine-readable storage medium 803, and optionallydevice units 804-808 that are interconnected via a bus or aninterconnect 810. A processor 801 may represent a single processor ormultiple processors with a single processor core or multiple processorcores included therein. The processor 801 may represent one or moregeneral-purpose processors such as a microprocessor, a centralprocessing unit (CPU), or processing device. More particularly, theprocessor 801 may be a complex instruction set computing (CISC)microprocessor, reduced instruction set computing (RISC) microprocessor,very long instruction word (VLIW) microprocessor, or processorimplementing other instruction sets, or processors implementing acombination of instruction sets. Processor 801 may also be one or morespecial-purpose processors such as an application specific integratedcircuit (ASIC), a cellular or baseband processor, a field programmablegate array (FPGA), a digital signal processor (DSP), a networkprocessor, a graphics processor, a network processor, a communicationsprocessor, a cryptographic processor, a co-processor, an embeddedprocessor, or any other type of logic capable of processinginstructions.

The processor 801 may communicate with non-transitory machine-readablestorage medium 803, which in an embodiment can be implemented viamultiple memory devices to provide for a given amount of system memory.The non-transitory machine-readable storage medium 803 may include oneor more volatile storage (or memory) devices such as random-accessmemory (RAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), static RAM(SRAM), or other types of storage devices. The non-transitorymachine-readable storage medium 803 may store information includingsequences of instructions that are executed by the processor 801, or anyother device units. For example, executable code and/or data of avariety of operating systems, device drivers, firmware (e.g., inputoutput basic system or BIOS), and/or applications can be loaded in thenon-transitory machine-readable storage medium 803 and executed by theprocessor 801. An operating system can be any kind of operating systems,such as, for example, Windows® operating system from Microsoft®, MacOS®/iOS® from Apple, Android® from Google®, Linux®, Unix®, or otherreal-time or embedded operating systems such as VxWorks.

The non-transitory machine-readable storage medium 803 contains thecamera application 222 to perform the operations discussed above.

The electronic device 800 may optionally further include input/output(I/O) devices such as the device units 804-808, including displaycontrol and/or display device unit 804, wireless transceiver(s) 805,video I/O device unit(s) 806, audio I/O device unit(s) 807, and otherI/O device units 808 as illustrated. The wireless transceiver 805 may bea WiFi transceiver, an infrared transceiver, a Bluetooth transceiver, aWiMax transceiver, a wireless cellular telephony transceiver, asatellite transceiver (e.g., a global positioning system (GPS)transceiver), or other radio frequency (RF) transceivers, or acombination thereof. The electronic device 800 may also include anultrasound device unit (not shown) for transmitting a conference sessioncode.

The video I/O device unit 806 may include an imaging processingsubsystem (e.g., a camera), which may include an optical sensor, such asa charged coupled device (CCD) or a complementary metal-oxidesemiconductor (CMOS) optical sensor, utilized to facilitate camerafunctions, such as recording photographs and video clips andconferencing. An audio I/O device unit 807 may include a speaker and/ora microphone to facilitate voice-enabled functions, such as voicerecognition, voice replication, digital recording, and/or telephonyfunctions. Other optional devices 808 may include a storage device(e.g., a hard drive, a flash memory device), universal serial bus (USB)port(s), parallel port(s), serial port(s), a printer, a networkinterface, a bus bridge, sensor(s) (e.g., a motion sensor such as anaccelerometer, gyroscope, a magnetometer, a light sensor, compass, aproximity sensor, etc.), or a combination thereof. The optional deviceunits 808 may further include certain sensors coupled to theinterconnect 810 via a sensor hub (not shown), while other devices suchas a keyboard or thermal sensor may be controlled by an embeddedcontroller (not shown), dependent upon the specific configuration ordesign of the electronic device 800.

Note that while the electronic device 800 is illustrated with variouscomponents, it is not intended to represent any particular architectureor manner of interconnecting the components; as such details are notgermane to embodiments of the present invention. It will also beappreciated that an electronic device having fewer components or perhapsmore components may also be used with embodiments of the invention.

Some portions of the preceding detailed descriptions have been presentedin terms of algorithms and symbolic representations of operations ondata bits within a computer memory. These algorithmic descriptions andrepresentations are the ways used by those skilled in conferencingtechnology to most effectively convey the substance of their work toothers skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of operations leading to adesired result. The operations are those requiring physicalmanipulations of physical quantities.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the above discussion, itis appreciated that throughout the description, discussions utilizingterms such as those set forth in the claims below, refer to the actionand processes of a conference device, or similar electronic computingdevice, that manipulates and transforms data represented as physical(electronic) quantities within the conference device's registers andmemories into other data similarly represented as physical quantitieswithin the conference device's memories or registers or other suchinformation storage, transmission or display devices.

Environment Utilizing Embodiments of the Invention

While the invention has been described in terms of several embodiments,those skilled in the art will recognize that the invention is notlimited to the embodiments described, can be practiced with modificationand alteration within the spirit and scope of the appended claims. Thedescription is thus to be regarded as illustrative instead of limiting.

What is claimed is:
 1. A camera system, comprising: a lens; and an imagesensor, wherein the camera system is rotatable about a single axis,wherein the image sensor has more pixel sensors in a first orientationsubstantially parallel to the single axis than in a second orientationperpendicular to the first orientation, and wherein an application,coupled to the camera system via a wireless network, is to controlrotation of the camera system.